Skip to content

feat(cli): cloudhal import reads a site archive (#47, step 3) - #131

Merged
Madraka merged 6 commits into
mainfrom
feat/issue-47-import
Oct 25, 2026
Merged

Madraka merged 6 commits into
mainfrom
feat/issue-47-import

Conversation

@Madraka

@Madraka Madraka commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

What

Step 3 of RFC-0039's rollout for #47: cloudhal import <file> [--dry-run] in bootstrap mode. Stacked on #130 (cloudhal export); the diff shrinks to this step's commits once #130 merges.

fix(core-ts): importSite checks everything before it writes; planImport

Today importSite (behind pull, the admin bundle route push uses, and now import) writes node bodies before its bootstrap conflict check, inserts snapshots before its 1000-statement cap, and parses bundle asset bodies in the media ledger after the node batch. A refused or crafted bundle can therefore leave part of itself behind. It is now plan-then-apply:

  • New planImport(to, bundle, options) runs every check and target read in the old order and writes nothing. It returns the nodes added, changed and removed, the snapshots and bodies to add, whether the project row is new, the media present/missing/of another size/taken down on the target, and the takedowns. With report: true it returns a bootstrap conflict and missing media instead of throwing them.
  • New checks, all before the first write:
    • Every node body the target lacks must be a valid contract document of the kind, id and rev its snapshot refs name, belonging to the bundle's site. This uses the precompiled validators (RFC invariant 2). Before, a crafted bundle could put invalid content behind the published snapshot.
    • Unreferenced bodies, ids held by another site, duplicate ids, and kind/key or translation group/locale collisions are refused.
    • Malformed takedowns are refused.
  • The media ledger reads verified bodies only.
  • sync.ts (658 lines) is split into sync-bundle (types), sync-checks, sync-plan and sync-ledger. Every export of @cloudhal/core-ts stays.

feat(cli): cloudhal import

  • Pass 1 verifies the whole archive: sizes, SHA-256s, rows and node bodies. It applies the site rule: the archive's site must be cloudhal.json's siteId or be listed in sites; any other id is refused, and the switch comes with --replace in step 4. It then plans against the local store. Every refusal comes before any write, and a project without a store gets none created.
  • Pass 2 re-reads the file and refuses it if it changed. It copies only the missing media (FsBlobStore verifies them again), then calls importSite, keeping the archive's published snapshot as pull does.
  • --dry-run prints the manifest and what would be added, changed or removed: nodes by kind and title, snapshots, and media to copy. It exits 1 when the import would be refused.
  • Takedowns:
    • Taken-down content stays down.
    • A local copy of content the archive's ledger took down is deleted and reported.
    • An archive that carries media this project took down is refused.
  • Ctrl-C stops before the graph is written. A storage failure partway says so, and running the import again completes it, search index included.

Verification (local)

  • packages/core-ts: the full suite passes, 55 files and 997 tests, including the sqlite and Miniflare D1 behavioural suites that call importSite. The new sync-plan.test.ts has 9 tests covering the refusals with nothing written.
  • packages/cli: import-cli (25), export-cli (16), archive-* and cli-commands all pass, 182 tests in one process. That includes:
    • crafted bodies refused by both --dry-run and import;
    • the pass-2 guard when the file changes, which a mutation check confirmed;
    • a re-import after a takedown;
    • a storage failure partway, then completed by a second run.
  • Real data:
    • cloudhal.com's local copy (16 nodes, 6 snapshots, 46 historical bodies): export then import --dry-run into an empty project validates every historical body.
    • examples/ui-audit copies (24 snapshots, media): import into an empty store, a second run that is a no-op, a refusal after a local edit, and re-export gives identical bytes.
  • Typecheck (core-ts, cli, api), declarations, Biome and pnpm size:check pass.
  • Note: a first push or pull into an empty target now validates every historical body against today's schemas. Schemas only grow optional fields (AGENTS.md rule 3), and the real data above passes. CI's Cloudflare push/pull suites cover the route.

Checklist

  • pnpm check and cargo test --workspace pass locally. The core-ts suite was run in full; the CLI pieces that cover this change were run. There is no Rust or contract change, and CI runs the rest.
  • Contract change? No.
  • User-visible change in a published package? Yes: .changeset/import-plan.md (core-ts) and .changeset/site-archive-import.md (cli). The step-2 changesets lost their "import comes later" wording.
  • Commits are signed off (git commit -s, DCO)
  • Docs / AGENTS.md updated if behaviour or commands changed. help.ts is updated; the guide comes in step 5.

Refs #47

🤖 Generated with Claude Code

Madraka and others added 6 commits October 26, 2026 00:42
importSite inserted bundle.project as given when the target lacked it,
so a crafted bundle could plant any projects row. The row must now be the
site node's project, a valid Project, with id and rev matching its body.
Covers pull, the admin bundle route behind push, and the coming
`cloudhal import`.

Refs #47

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Madraka <[email protected]>
RFC-0039 rollout step 2. `cloudhal export [--out <file>] [--site <id>]
[--no-media] [--latest-only]` writes one site of the local project to a
.cloudhal file: exportSite's bundle as NDJSON parts, media streamed from
the local blob store and verified, and the takedown ledger.

- Default name <site-slug>-<YYYYMMDD>.cloudhal, dated by the newest row
  it holds (never the clock), so an unchanged site exports identically.
  A file at the default name is replaced only by the same archive; --out
  replaces any.
- --site takes a site listed in cloudhal.json sites; --site, --no-media
  and --latest-only are refused on every other command.
- --latest-only keeps the published and the latest snapshot and what
  they need; refused while a field operation is unfinished.
- Ctrl-C and SIGTERM remove the partial file.
- The summary lists what an archive never holds (RFC-0039 §4) and that
  its history names who made each change.

archive-bundle.ts and archive-reassemble.ts convert a bundle to archive
content and back (step 3's import reuses the second); the tests check
the round trip deep-equals exportSite's bundle.

Refs #47

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Madraka <[email protected]>
importSite wrote node bodies before its bootstrap conflict check,
snapshots before its 1000-statement cap, and parsed bundle asset bodies
in the media ledger after the node batch, so a refused or crafted bundle
could leave part of itself behind. It now plans first and applies after:

- planImport runs every check and target read in the old order and
  returns what an import would change (nodes added/changed/removed,
  snapshots and bodies to add, media present/missing/other size/taken
  down, takedowns); with report: true it returns a bootstrap conflict
  and missing media instead of throwing.
- Every node body the target lacks must be a valid contract document of
  the kind, id and rev its snapshot refs name, of the bundle's site; an
  unreferenced body is refused.
- Ids of another site, duplicate ids, kind/key and translation
  group/locale collisions, and malformed takedowns are refused up front.
- The media ledger reads verified bodies only.

sync.ts is split into sync-bundle (types), sync-checks, sync-plan and
sync-ledger; every export of @cloudhal/core-ts stays.

Refs #47

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Madraka <[email protected]>
RFC-0039 rollout step 3: `cloudhal import <file> [--dry-run]`, bootstrap
mode. Pass 1 verifies the whole archive (sizes, SHA-256s, rows, node
bodies), applies the site rule (the archive's site must be cloudhal.json's
siteId or listed in sites) and plans against the local store with
planImport; any refusal comes before a write, and a project without a
store gets none. Pass 2 re-reads the file, refuses it if it changed,
copies only the missing media and calls importSite, keeping the
archive's published snapshot as pull does.

--dry-run prints the manifest and what would be added, changed or
removed (nodes by kind and title, snapshots, media) and exits 1 when the
import would be refused. Takedowns stay down; local copies of content
the archive took down are deleted and reported. Ctrl-C stops before the
graph is written. --replace is refused as not yet available (step 4).

Also fixes the export test that picked a tied snapshot that was not the
(created_at, id) maximum.

Refs #47

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Madraka <[email protected]>
… test

The seed's clock moves in 20 ms steps, so other snapshots can already
share the newest one's created_at; the test picked the first earlier row
with a larger id instead of the largest, and failed when another tied row
sorted after it (CI on #130).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Madraka <[email protected]>
@Madraka
Madraka enabled auto-merge October 25, 2026 23:34
@Madraka
Madraka merged commit e8a558f into main Oct 25, 2026
5 checks passed
@Madraka
Madraka deleted the feat/issue-47-import branch October 25, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant